{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7a5d459f",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/uptrain-ai/uptrain/blob/main/examples/checks/conversation/query_resolution.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f74cbf3-07b5-40d8-86cd-6fe807d8f36e",
   "metadata": {
    "tags": []
   },
   "source": [
    "<h1 align=\"center\">\n",
    "  <a href=\"https://uptrain.ai\">\n",
    "    <img width=\"300\" src=\"https://user-images.githubusercontent.com/108270398/214240695-4f958b76-c993-4ddd-8de6-8668f4d0da84.png\" alt=\"uptrain\">\n",
    "  </a>\n",
    "</h1>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9fc0b549-d081-442e-87ce-1ed0657540f2",
   "metadata": {},
   "source": [
    "<h1 style=\"text-align: center;\">Evaluating Query Resolution in Conversations</h1>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f92208d-e85c-4108-951d-4f54940ace43",
   "metadata": {
    "tags": []
   },
   "source": [
    "**What is Query Resolution?**: Query resolution is the process of understanding the user's query and providing an appropriate response. In the context of a conversational AI, query resolution is the process of understanding the user's query and providing an appropriate response.\n",
    "\n",
    "For example, a user might ask a conversational AI, \"What is the weather like today?\" The conversational AI should understand that the user is asking for information about the weather and provide a response with the current weather conditions.\n",
    "\n",
    "**Data schema**: The data schema required for this evaluation is as follows:\n",
    "\n",
    "| Column Name | Description |\n",
    "| ----------- | ----------- |\n",
    "| conversation | The conversation between the user and the LLM |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82c5b58f-bfb9-4c63-8b22-8ed05fa1fa4d",
   "metadata": {},
   "source": [
    " If you face any difficulties, need some help with using UpTrain or want to brainstorm on custom evaluations for your use-case, [speak to the maintainers of UpTrain here](https://calendly.com/uptrain-sourabh/30min).\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f936a36",
   "metadata": {},
   "source": [
    "## Step 1: Install UpTrain by running 'pip install uptrain'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "37a5ab48",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: uptrain in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (0.7.0)\n",
      "Requirement already satisfied: pydantic in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (2.6.4)\n",
      "Requirement already satisfied: pydantic-settings in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (2.2.1)\n",
      "Requirement already satisfied: loguru in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (0.7.0)\n",
      "Requirement already satisfied: lazy-loader in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (0.3)\n",
      "Requirement already satisfied: networkx in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (3.1)\n",
      "Requirement already satisfied: polars>=0.18 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (0.20.13)\n",
      "Requirement already satisfied: pandas in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (2.0.3)\n",
      "Requirement already satisfied: numpy>=1.23.0 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (1.26.3)\n",
      "Requirement already satisfied: httpx>=0.24.1 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (0.26.0)\n",
      "Requirement already satisfied: plotly>=5.0.0 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (5.15.0)\n",
      "Requirement already satisfied: aiolimiter>=1.1 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (1.1.0)\n",
      "Requirement already satisfied: openai>=1.6.1 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (1.6.1)\n",
      "Requirement already satisfied: fsspec in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (2023.6.0)\n",
      "Requirement already satisfied: litellm in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (1.34.25)\n",
      "Requirement already satisfied: pyyaml in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (6.0.1)\n",
      "Requirement already satisfied: json5 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (0.9.24)\n",
      "Requirement already satisfied: python-docx in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (1.1.0)\n",
      "Requirement already satisfied: pymupdf in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (1.23.25)\n",
      "Requirement already satisfied: faiss-cpu in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (1.7.4)\n",
      "Requirement already satisfied: anyio in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from httpx>=0.24.1->uptrain) (3.7.1)\n",
      "Requirement already satisfied: certifi in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from httpx>=0.24.1->uptrain) (2023.7.22)\n",
      "Requirement already satisfied: httpcore==1.* in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from httpx>=0.24.1->uptrain) (1.0.2)\n",
      "Requirement already satisfied: idna in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from httpx>=0.24.1->uptrain) (3.4)\n",
      "Requirement already satisfied: sniffio in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from httpx>=0.24.1->uptrain) (1.3.0)\n",
      "Requirement already satisfied: h11<0.15,>=0.13 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from httpcore==1.*->httpx>=0.24.1->uptrain) (0.14.0)\n",
      "Requirement already satisfied: distro<2,>=1.7.0 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from openai>=1.6.1->uptrain) (1.8.0)\n",
      "Requirement already satisfied: tqdm>4 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from openai>=1.6.1->uptrain) (4.66.2)\n",
      "Requirement already satisfied: typing-extensions<5,>=4.7 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from openai>=1.6.1->uptrain) (4.7.1)\n",
      "Requirement already satisfied: tenacity>=6.2.0 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from plotly>=5.0.0->uptrain) (8.2.3)\n",
      "Requirement already satisfied: packaging in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from plotly>=5.0.0->uptrain) (23.2)\n",
      "Requirement already satisfied: annotated-types>=0.4.0 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from pydantic->uptrain) (0.6.0)\n",
      "Requirement already satisfied: pydantic-core==2.16.3 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from pydantic->uptrain) (2.16.3)\n",
      "Requirement already satisfied: aiohttp in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from litellm->uptrain) (3.9.1)\n",
      "Requirement already satisfied: click in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from litellm->uptrain) (8.1.6)\n",
      "Requirement already satisfied: importlib-metadata>=6.8.0 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from litellm->uptrain) (6.8.0)\n",
      "Requirement already satisfied: jinja2<4.0.0,>=3.1.2 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from litellm->uptrain) (3.1.2)\n",
      "Requirement already satisfied: python-dotenv>=0.2.0 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from litellm->uptrain) (1.0.0)\n",
      "Requirement already satisfied: requests<3.0.0,>=2.31.0 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from litellm->uptrain) (2.31.0)\n",
      "Requirement already satisfied: tiktoken>=0.4.0 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from litellm->uptrain) (0.4.0)\n",
      "Requirement already satisfied: tokenizers in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from litellm->uptrain) (0.15.1)\n",
      "Requirement already satisfied: python-dateutil>=2.8.2 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from pandas->uptrain) (2.8.2)\n",
      "Requirement already satisfied: pytz>=2020.1 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from pandas->uptrain) (2023.3)\n",
      "Requirement already satisfied: tzdata>=2022.1 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from pandas->uptrain) (2023.3)\n",
      "Requirement already satisfied: PyMuPDFb==1.23.22 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from pymupdf->uptrain) (1.23.22)\n",
      "Requirement already satisfied: lxml>=3.1.0 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from python-docx->uptrain) (4.9.3)\n",
      "Requirement already satisfied: zipp>=0.5 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from importlib-metadata>=6.8.0->litellm->uptrain) (3.16.2)\n",
      "Requirement already satisfied: MarkupSafe>=2.0 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from jinja2<4.0.0,>=3.1.2->litellm->uptrain) (2.1.3)\n",
      "Requirement already satisfied: six>=1.5 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from python-dateutil>=2.8.2->pandas->uptrain) (1.16.0)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from requests<3.0.0,>=2.31.0->litellm->uptrain) (3.2.0)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from requests<3.0.0,>=2.31.0->litellm->uptrain) (1.26.16)\n",
      "Requirement already satisfied: regex>=2022.1.18 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from tiktoken>=0.4.0->litellm->uptrain) (2023.8.8)\n",
      "Requirement already satisfied: attrs>=17.3.0 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from aiohttp->litellm->uptrain) (23.1.0)\n",
      "Requirement already satisfied: multidict<7.0,>=4.5 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from aiohttp->litellm->uptrain) (6.0.4)\n",
      "Requirement already satisfied: yarl<2.0,>=1.0 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from aiohttp->litellm->uptrain) (1.9.2)\n",
      "Requirement already satisfied: frozenlist>=1.1.1 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from aiohttp->litellm->uptrain) (1.4.0)\n",
      "Requirement already satisfied: aiosignal>=1.1.2 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from aiohttp->litellm->uptrain) (1.3.1)\n",
      "Requirement already satisfied: huggingface_hub<1.0,>=0.16.4 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from tokenizers->litellm->uptrain) (0.20.3)\n",
      "Requirement already satisfied: filelock in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from huggingface_hub<1.0,>=0.16.4->tokenizers->litellm->uptrain) (3.12.2)\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "%pip install uptrain"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb91700c",
   "metadata": {},
   "source": [
    "## Step 2: Let's define our dataset to run evaluations upon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "bf4c2b81-59f4-4faa-b474-7b3f4c2ed509",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "satisfactory_chat = [{\n",
    "    'conversation' : [\n",
    "        {\"role\": \"patient\", \"content\": \"Help\"}, \n",
    "        {\"role\": \"nurse\", \"content\": \"what do you need\"}, \n",
    "        {\"role\": \"patient\", \"content\": \"Having chest pain\"}, \n",
    "        {\"role\": \"nurse\", \"content\": \"Do you have any other symptoms?\"},\n",
    "        {\"role\": \"patient\", \"content\": \"I am having difficulty breathing\"},\n",
    "        {\"role\": \"nurse\", \"content\": \"I will call the doctor right away\"},\n",
    "        {\"role\": \"patient\", \"content\": \"Thank you\"}\n",
    "    ]  \n",
    "}]\n",
    "\n",
    "unsatisfactory_chat = [{\n",
    "    'conversation' : [\n",
    "        {\"role\": \"patient\", \"content\": \"Help\"}, \n",
    "        {\"role\": \"nurse\", \"content\": \"what do you need\"}, \n",
    "        {\"role\": \"patient\", \"content\": \"Having chest pain\"}, \n",
    "        {\"role\": \"nurse\", \"content\": \"Sorry, I am not sure what that means\"},\n",
    "        {\"role\": \"patient\", \"content\": \"You don't understand. Do something! I am having severe pain in my chest\"}\n",
    "    ]  \n",
    "}]\n",
    "\n",
    "data = satisfactory_chat + unsatisfactory_chat"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b640784e",
   "metadata": {},
   "source": [
    "## Step 3: Running evaluations using UpTrain's Open-Source Software (OSS)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "d6706deb-8e9d-4d45-8f9b-aed0e9da608f",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages/lazy_loader/__init__.py:185: RuntimeWarning: subpackages can technically be lazily loaded, but it causes the package to be eagerly loaded even if it is already lazily loaded.So, you probably shouldn't use subpackages with this lazy feature.\n",
      "  warnings.warn(msg, RuntimeWarning)\n",
      "\u001b[32m2024-05-17 16:45:43.434\u001b[0m | \u001b[33m\u001b[1mWARNING \u001b[0m | \u001b[36muptrain.operators.language.llm\u001b[0m:\u001b[36mfetch_responses\u001b[0m:\u001b[36m268\u001b[0m - \u001b[33m\u001b[1mDetected a running event loop, scheduling requests in a separate thread.\u001b[0m\n",
      "100%|██████████| 2/2 [00:01<00:00,  1.06it/s]\n",
      "/Users/dhruvchawla/work/uptrain-v1/uptrain/operators/language/llm.py:271: RuntimeWarning: coroutine 'LLMMulticlient.async_fetch_responses' was never awaited\n",
      "  with ThreadPoolExecutor(max_workers=1) as executor:\n",
      "RuntimeWarning: Enable tracemalloc to get the object allocation traceback\n"
     ]
    }
   ],
   "source": [
    "from uptrain import EvalLLM, QueryResolution\n",
    "import json\n",
    "\n",
    "OPENAI_API_KEY = \"sk-******************************\"  # Insert your OpenAI key here\n",
    "\n",
    "eval_llm = EvalLLM(openai_api_key=OPENAI_API_KEY)\n",
    "\n",
    "res = eval_llm.evaluate(\n",
    "    data = data,\n",
    "    checks = [QueryResolution(user_persona=\"patient\", llm_persona=\"nurse\")],\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "fa5df117-5516-4e32-b86b-e3ade08a5d78",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[\n",
      "   {\n",
      "      \"conversation\": [\n",
      "         {\n",
      "            \"role\": \"patient\",\n",
      "            \"content\": \"Help\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"nurse\",\n",
      "            \"content\": \"what do you need\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"patient\",\n",
      "            \"content\": \"Having chest pain\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"nurse\",\n",
      "            \"content\": \"Do you have any other symptoms?\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"patient\",\n",
      "            \"content\": \"I am having difficulty breathing\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"nurse\",\n",
      "            \"content\": \"I will call the doctor right away\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"patient\",\n",
      "            \"content\": \"Thank you\"\n",
      "         }\n",
      "      ],\n",
      "      \"score_query_resolution\": 1.0,\n",
      "      \"explanation_query_resolution\": \"{\\n    \\\"Reasoning\\\": \\\"The nurse quickly identifies the severity of the patient's symptoms (chest pain and difficulty breathing) and takes immediate action by deciding to call the doctor right away. This shows that the nurse is addressing the patient's urgent medical needs and providing appropriate assistance. The query is effectively resolved.\\\",\\n    \\\"Choice\\\": \\\"A\\\"\\n}\"\n",
      "   },\n",
      "   {\n",
      "      \"conversation\": [\n",
      "         {\n",
      "            \"role\": \"patient\",\n",
      "            \"content\": \"Help\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"nurse\",\n",
      "            \"content\": \"what do you need\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"patient\",\n",
      "            \"content\": \"Having chest pain\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"nurse\",\n",
      "            \"content\": \"Sorry, I am not sure what that means\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"patient\",\n",
      "            \"content\": \"You don't understand. Do something! I am having severe pain in my chest\"\n",
      "         }\n",
      "      ],\n",
      "      \"score_query_resolution\": 0.0,\n",
      "      \"explanation_query_resolution\": \"{\\n    \\\"Reasoning\\\": \\\"The nurse initially asks the patient what they need, but when the patient mentions having chest pain, the nurse responds by saying they are not sure what that means. This shows a lack of understanding and empathy towards the patient's situation. The patient then clearly expresses the severity of their pain, but the nurse still does not provide any helpful response or take appropriate action.\\\",\\n    \\\"Choice\\\": \\\"C\\\"\\n}\"\n",
      "   }\n",
      "]\n"
     ]
    }
   ],
   "source": [
    "print(json.dumps(res,indent=3))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49ea84a4-2846-4a3c-97c2-ac8c8d413c52",
   "metadata": {},
   "source": [
    "## Step 4: Let's look at some of the results "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db249e6d-5a30-43aa-92f0-6217f6ca85f3",
   "metadata": {},
   "source": [
    "### Sample with a good conversation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "6343e0c0-35a1-4d59-a771-dd8a0116fb50",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "   \"conversation\": [\n",
      "      {\n",
      "         \"role\": \"patient\",\n",
      "         \"content\": \"Help\"\n",
      "      },\n",
      "      {\n",
      "         \"role\": \"nurse\",\n",
      "         \"content\": \"what do you need\"\n",
      "      },\n",
      "      {\n",
      "         \"role\": \"patient\",\n",
      "         \"content\": \"Having chest pain\"\n",
      "      },\n",
      "      {\n",
      "         \"role\": \"nurse\",\n",
      "         \"content\": \"Do you have any other symptoms?\"\n",
      "      },\n",
      "      {\n",
      "         \"role\": \"patient\",\n",
      "         \"content\": \"I am having difficulty breathing\"\n",
      "      },\n",
      "      {\n",
      "         \"role\": \"nurse\",\n",
      "         \"content\": \"I will call the doctor right away\"\n",
      "      },\n",
      "      {\n",
      "         \"role\": \"patient\",\n",
      "         \"content\": \"Thank you\"\n",
      "      }\n",
      "   ],\n",
      "   \"score_query_resolution\": 1.0,\n",
      "   \"explanation_query_resolution\": \"{\\n    \\\"Reasoning\\\": \\\"The nurse quickly identifies the severity of the patient's symptoms (chest pain and difficulty breathing) and takes immediate action by deciding to call the doctor right away. This shows that the nurse is addressing the patient's urgent medical needs and providing appropriate assistance. The query is effectively resolved.\\\",\\n    \\\"Choice\\\": \\\"A\\\"\\n}\"\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "print(json.dumps(res[0],indent=3))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a7388f1",
   "metadata": {},
   "source": [
    "### Sample with a bad conversation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "9c1a6497",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "   \"conversation\": [\n",
      "      {\n",
      "         \"role\": \"patient\",\n",
      "         \"content\": \"Help\"\n",
      "      },\n",
      "      {\n",
      "         \"role\": \"nurse\",\n",
      "         \"content\": \"what do you need\"\n",
      "      },\n",
      "      {\n",
      "         \"role\": \"patient\",\n",
      "         \"content\": \"Having chest pain\"\n",
      "      },\n",
      "      {\n",
      "         \"role\": \"nurse\",\n",
      "         \"content\": \"Sorry, I am not sure what that means\"\n",
      "      },\n",
      "      {\n",
      "         \"role\": \"patient\",\n",
      "         \"content\": \"You don't understand. Do something! I am having severe pain in my chest\"\n",
      "      }\n",
      "   ],\n",
      "   \"score_query_resolution\": 0.0,\n",
      "   \"explanation_query_resolution\": \"{\\n    \\\"Reasoning\\\": \\\"The nurse initially asks the patient what they need, but when the patient mentions having chest pain, the nurse responds by saying they are not sure what that means. This shows a lack of understanding and empathy towards the patient's situation. The patient then clearly expresses the severity of their pain, but the nurse still does not provide any helpful response or take appropriate action.\\\",\\n    \\\"Choice\\\": \\\"C\\\"\\n}\"\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "print(json.dumps(res[1],indent=3))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2904e2f4",
   "metadata": {},
   "source": [
    "## [Optional] Step 5: UpTrain Managed Service and Dashboards"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0bb801c3",
   "metadata": {},
   "source": [
    "You can create a free UpTrain account [here](https://uptrain.ai/) and get free trial credits. If you want more trial credits, [book a call with the maintainers of UpTrain here](https://calendly.com/uptrain-sourabh/30min)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "5e23c4a7",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m2024-05-17 16:45:45.483\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36muptrain.framework.remote\u001b[0m:\u001b[36mlog_and_evaluate\u001b[0m:\u001b[36m677\u001b[0m - \u001b[1mSending evaluation request for rows 0 to <50 to the Uptrain server\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[\n",
      "   {\n",
      "      \"conversation\": [\n",
      "         {\n",
      "            \"role\": \"patient\",\n",
      "            \"content\": \"Help\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"nurse\",\n",
      "            \"content\": \"what do you need\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"patient\",\n",
      "            \"content\": \"Having chest pain\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"nurse\",\n",
      "            \"content\": \"please call 102\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"patient\",\n",
      "            \"content\": \"Thank you nurse\"\n",
      "         }\n",
      "      ],\n",
      "      \"score_query_resolution\": 1.0,\n",
      "      \"explanation_query_resolution\": \"The user initially asked for help, followed by mentioning chest pain. The nurse's response to call 102 seems appropriate for a medical emergency like chest pain. The user's final response indicates gratitude, showing satisfaction with the assistance provided.\"\n",
      "   },\n",
      "   {\n",
      "      \"conversation\": [\n",
      "         {\n",
      "            \"role\": \"patient\",\n",
      "            \"content\": \"Help\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"nurse\",\n",
      "            \"content\": \"what do you need\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"patient\",\n",
      "            \"content\": \"Having chest pain\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"nurse\",\n",
      "            \"content\": \"Sorry, I am not sure what that means\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"patient\",\n",
      "            \"content\": \"You don't understand. Do something! I am having severe pain in my chest\"\n",
      "         }\n",
      "      ],\n",
      "      \"score_query_resolution\": 0.0,\n",
      "      \"explanation_query_resolution\": \"The AI assistant failed to effectively address the patient's urgent concern of chest pain, which could indicate a potential medical emergency. The responses provided by the AI assistant did not offer appropriate assistance or guidance in such a critical situation.\"\n",
      "   }\n",
      "]\n"
     ]
    }
   ],
   "source": [
    "from uptrain import APIClient, Settings, QueryResolution\n",
    "\n",
    "UPTRAIN_API_KEY = \"up-**************************\"  # Insert your UpTrain API key here\n",
    "\n",
    "uptrain_client = APIClient(\n",
    "    Settings(\n",
    "        uptrain_access_token=UPTRAIN_API_KEY,\n",
    "    )\n",
    ")\n",
    "\n",
    "res = uptrain_client.log_and_evaluate(\n",
    "    \"Emergency-Contact-Information\",\n",
    "    data=data, \n",
    "    checks=[QueryResolution(user_persona=\"patient\", llm_persona=\"nurse\")],\n",
    ")\n",
    "\n",
    "print(json.dumps(res, indent=3))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
